Digitally based multimedia, the combination of video and audio in a digital format for viewing on a digital device is rapidly increasing in capacity and proliferation. Nearly every new personal computer manufactured today includes some form of multimedia. Sales of digital products such as cameras, video recorders, phones and televisions are steadily increasing. Multimedia is also becoming increasingly prevalent in the Internet realm as the growth of the Internet steadily and rapidly continues. Along with this growth have come increased performance expectations by the users of such computer equipment. These increased user expectations extend not only to hardware capability, but also to the processing capability of the data itself.
A technique known as streaming has been developed for multimedia applications to satisfy these increasing expectations. Streaming allows data to be transferred so that it can be processed as a steady and continuous stream. This has the benefit that data can be displayed or listened to before the entire file has been transmitted, a must for large multimedia files.
Initially, the streaming framework consisted of a chain of data processing modules (e.g., capturing filters, transformation filters, and rendering filters) with very little intelligence from the chain manager. The data processing modules, also called filters, make the decisions on how to get connected, what data formats to use, and how to control each other. During connection of filters in a chain, protocols define a predefined fixed sequence of data flow and control connection negotiations. A typical negotiation sequence is to negotiate the following in order: the interface, the medium, the data format, the allocators, and the master clock. The data processing chain provides an end-to-end solution within a computer system.
As the complexity of streaming increased, the industry recognized that it became necessary to optimize the processing chains that were processing data with real-time constraints such as video and audio processing chains. One solution is DirectShow® by Microsoft® Corporation, which provides playback of multimedia streams from local files or Internet servers, capture of multimedia streams from devices, and format conversion of multimedia streams. DirectShow® enables playback of video and audio content of file types such as Windows Media Audio, Windows Media Video, MPEG, Apple® QuickTime®, Audio-Video Interleaved (AVI), and WAV (Windows Wave). DirectShow® includes a system of pluggable filter components. Filters are objects that support DirectShow® interfaces and that are capable of operating on streams of data by reading, copying, modifying and writing data to a file. The basic types of filters include a source filter, which takes the data from some source, such as a file on disk, a satellite feed, an Internet server, or a VCR, and introduces it into the filter graph, which is a connection of filters. The filters in a filter graph include a transform filter, which converts the format of the data, a sync and source filter that receives data and transmits the data; and a rendering filter, which renders the data, such as rendering the data to a display device. The data can also be rendered to any location that accepts media. Other types of filters included in DirectShow® include effect filters, which add effects without changing the data type, and parser filters, which understand the format of the source data and know how to read the correct bytes, create times stamps, and perform seeks.
During operation, all data passes from filter to filter along with a great amount of control information. Each filter contains objects called “pins” that are used to connect to other filters. When filters are connected using the pins, a filter graph is created. Note there is a distinction between a “filter graph” which is the concept of a group of connected filters, and a “Filter Graph” which is the object you create in DirectShow® that controls the group of connected filters, the “filter graph”. The “Filter Graph” is more correctly called a filter graph manager. To control the data flow and connections in a filter graph, DirectShow® includes a filter graph manager. The filter graph manager assists in assuring that filters are connected in the proper order. However, the data and much of the control do not pass through the filter graph manager. Filters must be linked appropriately. For example, the filter graph manager must search for a rendering configuration, determine the types of filters available, link the filters in the appropriate order for a given data type and provide an appropriate rendering filter.
While filters allowed a great deal of reuse of programs, the use of these filters also created some unanticipated problems. One of the problems created by the filters is the large number of API's for the filters that came into being. Each filter essentially has a separate API. As a result, a given filter must be capable of interfacing to the API for every filter to which it might attach. Also, the use of filters creates the problem of shutting down a given filter problematic. When a given filter in a graph is shut down, any filter that interfaces with the shut down filter requires a different associated interface. In general, programming a filter to gracefully handle the loss of an interface is difficult, as the state of the filter can be unknown when the interface is lost. The loss of interfaces, therefore, tends to lead to unpredicted behavior in the filters and ultimately to ill behaved programs. Further, the overall control in DirectShow® is distributed between two blocks. The interface between the filters controls the data flow while the filter manager controls the instantiation and removal of filters. Distributing the control in this manner makes software design cumbersome, as there are inevitably some control functions that cross the boundary between the blocks. Another problem with DirectShow is that the filters shoulder the responsibility of media format negotiation and buffer management functionality. Filters communicate with other filters to accomplish this task. The dependency on filters causes applications building on DirectShow to be susceptible to bugs and inefficiencies that could be programmed into a filter. Thus, a badly written filter could easily bring down the filter graph and an application associated with the filter graph.
There is a need to improve the DirectShow® architecture. More particularly, there is a need to improve control of processing of multimedia data.